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1. INTRODUCTION 

Usability is one of the software’s quality attributes that is so widely considered in the development 
of successful interactive software. Usability evaluation is a key process in the usability field that checks 
whether a software product, especially its user interface possesses all required usability aspects as well as the 
extent of these [1]. It tries to find out, through massive methods and techniques, whether the product is easy 
to learn and use, efficient and effective in achieving users’ goals, or helps the users to perform their tasks [2]. 
It is commonly used in development projects where usability is a primary concern [3]. 

Agile methodologies that emerged in the last decade provide an alternative to traditional approaches 
for building software, which are aimed at satisfying the customer in the software industry [4]. They seek to 
minimize the overall risk, enabling the project to be adapted with modifications rapidly, and maintaining the 
project schedule and budget [5]. They follow an iterative and incremental approach to developing software 
rapidly where the entire software development lifecycle is broken down into smaller iterations [6]. 

Up to the present, one of the main challenges in the agile software development community is how 
to incorporate usability evaluation work into the development process in an efficient and effective manner. 
Despite both fields are similar in many aspects, such as a shared focus on delivering value, being iterative in 
nature, and including continuous testing, they are different in how approach these aspects practically [7], [8]. 
Yet, many methods and techniques have been proposed for reconciling usability work with the agile 
development environment, but they have lacked a comprehensive strategy to persisting perform evaluations 
of usability alongside the development process [9]-[11]. Determining how to promote ongoing usability 
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evaluations in the agile development environment provides an interesting research opportunity, the results of 
which can be applied to improving the incorporation of usability evaluations’ work. 


2. INTEGRATION OF USABILITY EVALUATION INTO THE AGILE APPROACH 

Explaining Integrating usability evaluation into the agile development approach has been addressed 
as a means by which these tensions might be resolved. For instance, Metzker and Reiterer presented the 
approach "evidence-based computer-aided usability engineering environment (CAUSE)" which covers the 
end-user aspects, where agile methods such as extreme programming (XP), feature driven development 
(FDD) or Scrum were originally developed to focus on satisfying development and business needs rather 
than end-user needs [12]. Diichting et al. [13] claim that existing agile approaches do not offer specific 
guidance on how to integrate usability practices into agile activities. Because these approaches started as 
methods for programming, development practitioners are often uncertain about how non-coders fit into the 
agile development team. They discussed the application of usability evaluation works in the agile 
development environment and drawn practice-oriented recommendations. Chamberlain et al. [14] carried out 
a case study to ground a broad framework about how usability and agile approaches can work together. The 
case study shows that this goal can be achieved through exploit typical values and practices from both 
approaches. Also, the study highlights the efforts that have to be made to ensure appropriate resource 
management, balance, and participation. Based on practical experiences, Parsons ef al. [15] argue that 
because usability evaluation stems from a different area, no typical methods, techniques, or tools can be 
integrated directly into the agile methods, which require adapted practices for this integration. Among other 
integration strategies, Lievesley claims that adding an interaction designer role within the development team 
could contribute to observing, steering, and managing the activities of usability [16]. Sy and Miller [17] 
claim that despite the production of working software at a quick and constant pace providing a great setting 
for usability evaluations, traditional usability evaluations conducted in a laboratory are hardly fitting this kind 
of process. They recommended that for achieving usable software, the user interaction design must be 
considered an essential part of software development. 

Magiies et al. [18] proposed an approach that parallels the process between interaction design and 
coding. Through careful coordination, the interaction designers do an elaborated design. In light of this 
design, developers produce working software. Next, testers evaluate the working software using lightweight 
evaluation methods. Ardito et al. [19] argue that usability issues are important to developing an interactive 
software product in that agile and usability practitioners need to conform for an effective integration. They 
suggested recommendations aimed at understanding and learning usability methods and techniques and 
accepting that the concept of usability is an important quality aspect. Zorzetti et al. [20] performed a case 
study to investigate the incorporation of usability methods within an agile process using multi techniques, 
and they claim that a software development process model that encompasses Agile, Lean Startup, and User- 
Centered Design concepts can be used as a starting point for those who want to adhere to such a development 
approach. Gardner and Ozgur [21] propose a development framework that champions early usability 
integration in an Agile environment by analyzing their effect on the Scrum and Kanban development 
frameworks. Among the processes explored are the participatory design processes, joint-application 
development (JAD) sessions and design thinking activities, ethnographic studies such as contextual task 
analysis or inquiry, and those that modify the iterative process with the intent of prioritizing analysis and 
design such as the use of design sprints. Also, they evaluated the components of the proposed framework 
during the requirements gathering, analysis, and design phases of development by interview technique [21]. 

However, user interface design and evaluation are not a simple endeavor for systems that have a large 
UI component. Moreover, usability activities are time-consuming and may violate any of the accepted tenets of 
the agile development philosophy. Therefore, applying usability evaluations at the right place and at the right 
time is required to ensure both agility plus continuous evaluations alongside the development process. 


3. THE MODEL FOR SELECTING SUITABLE METHODS OF USABILITY EVALUATION 

Due to the short iteration time of agile development, the developer is more likely to sacrifice 
usability work to get the code implemented since functioning code is central to agile methods [22]. In 
practice, many agile processes have a narrow view of what competencies are needed in a project, and 
usability activities are routinely overlooked. Without these activities the chances of producing usable systems 
are slight. This is a serious weakness affecting most conventional agile methodologies. Where, if usability 
evaluation was considered within the development priorities, the agile values could be used to promote the 
idea that agile methods should be empowered, multidisciplinary, and include usability evaluation activities. 
To achieve this target, the authors of this paper propose a model for the selection and application of 
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appropriate usability evaluation methods and techniques in the right places at the right times based on certain 
criteria. Due to the proposed model tends to be used by the agile development team, terminologies are 
created familiar to the development environment, and the characterization of the methods was taken from an 
agile perspective. The development team should select suitable evaluations of usability to be within the 
development activities. For example, selecting methods that require less training for an agile team and 
techniques that improve the product's usability relative to the application effort. The following is a list of 
characteristics, metrics, and measurements for selecting suitable evaluation methods (see Figure 1). 

The model acts as a flexible instrument so that the implementation of a specific procedure or process 
is not necessary. Since iterative refinement is a required characteristic of every user-centered development 
endeavor, the sole requirement for the current development process is that it be based on it. 


Metric 
Measure 


Scale 


Figure 1. The general model of usability evaluation methods 


4. SELECTION CRITERIA 

The proposed model provides a set of attributes for selecting appropriate evaluation methods to 
apply during software development phases (see Figure 2). Some of these attributes were proposed by [23], 
the authors involve further attributes such as satisfaction, understandability, and attitude. Despite the 
automation attribute represented in the previous set, it is found far from affecting selection usability methods. 
For this, it is omitted from the proposed set. Learnability also is adapted within the set, which in this model 
learnability represents the ease with which the method required for achieving particular goals can be 
mastered, rather than how easy it is to learn the method. Table 1 shows these attributes and describes each 
one from the agile development perspective such as short-time iterations, high-level of collaboration with 
customers, focus on working artifacts, and dynamic processes. 


Selection of usability evaluation 
methods 


Effectiveness Dynamicity Understandabili Learnability Results 
Accuracy 
Coverage Area Time-cost | Effort-cost | | Attitude Satisfaction 


Figure 2. Usability evaluation model for selecting suitable methods 


Table 1. Set of attributes for selection suitable usability evaluation methods 


Attributes Description 
Effectiveness Feedback effects on design or development 
Dynamicity The ability of the method to be changed according to the target environment 
Understandability | Whether the user can understand the functions and usage of software easily (e.g., through familiar models, and metaphors). 
Learnability The ease with which the method required for achieving particular goals can be mastered. It is the capability of the 


software product to enable users to feel that they can productively use the software product right away and then 
quickly learn other new (for them) functionalities. 
Results accuracy Accuracy of results. 


Coverage area The elicitation/usability issues are covered by this method. 
Time-cost How much minimum time is needed to complete this method. 
Effort-cost How much effort is needed to perform this method (e.g., manpower, equipment, experiment place, other resources). 
Attitude The attitude of users or their subjective feelings during the use of the product. 
Satisfaction Freedom from discomfort and positive attitudes towards the use of the product. In other words, the subjective 


responses from users about their feelings when using the software (i.e., is the user satisfied or happy with the system?). 
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The effectiveness attribute is to assess the method’s impact on designing and/or development. A 
method with high impact gets more priority than the lower one, like heuristic evaluation with usability 
experts on early prototypes can have more impact on design improvements compared to other evaluation 
methods. The agile processes can adopt the target-working environment; therefore, methods with higher 
degrees of dynamicity are natural alliance. The dynamicity attribute is to judge a method’s ability to change 
its process of working according to the target environment. The time-cost attribute is to determine the time 
needed to perform a particular usability evaluation method. This has a direct relation with the time frame of 
agile iteration. Therefore, we suggest choosing those methods that can fit properly in the time frame of the 
targeted iteration. 

The understandability attribute talks about the understandability of usability evaluation methods by 
the responsible persons within the development team. A method with higher ease of understandability takes 
less time while performing it, hence compliment to agile short nature of iterations. The satisfaction attribute 
talks about freedom from discomfort and satisfaction towards the use of the method in term of effectiveness, 
correctness, and verifiability. The attitude talks about subjective feelings of the responsible persons during 
the use of the method. As such, a method with higher positive attitudes takes less time while performing it, 
hence complementing agile’s short nature of iterations. Also, a method’s results act as feedback for the 
improvements in the design and development of the developed product. 

The effort-cost talks about other resources (manpower, equipment, money, and experiment place) 
that are needed to perform the selected method. This also has a direct relation with the targeted agile 
iteration. For example, normally agile teams are made up with a small set of people and a particular method 
may not be well suited if it requires more man power than the team’s capability. The learnability attribute 
talks about a usability evaluation method’s learnability. A method with greater ease of learnability takes less 
time while performing it, which makes it easy to obtain information about improving product usability. 

The accuracy feedback of a particular method can play a critical role in the successful outcomes of 
agile development iteration. Each usability evaluation method covers only parts of the problem. The coverage 
area describes the perspective of highlighting eliciting issues or usability issues by a particular method. 
Hence, it is better to select those two methods that highlight issues from different perspectives rather than 
those two from the same perspective. 

A central aim of this model is to apply evaluations of usability smoothly during the agile process. 
Using the selection attributes of the model would enable agile practitioners to specify which usability 
evaluation methods are suitable and possible to apply in the right places at the right times. The contexts in 
which these methods are appropriate, the reasons why they should be employed, the data they yield, in what 
type of organization or for what purpose they would be most valuable, value each offers, and the downsides 
of each. However, with using this model the authors recommend that for early design activities, an emphasis 
on paper-based or simple UI prototype-based evaluation methods to improve design early. In later iteration 
activities, using formal evaluation methods, such as task model-based usability evaluation method, to get 
formal results. Also, the authors recommend using a mixture of evaluation methods, preferably supported by 
end users and UX experts for maximum results. 


5. METRICS 

The usability literature has offered a rather large number of testing metrics. These metrics fall into 
two categories: (i) preference metrics, which quantify subjective evaluations, and (ii) performance metrics, 
which evaluate the actual performance of the users when completing a task in a certain scenario (e.g. success 
rate, error rate, time completion time). 

We have discovered a total of 18 metrics for usability evaluation attributes based on research of 
existing usability measurement standards and frameworks (see Table 2). These metrics are derived from the 
agile software development perspective, which categorizes usability evaluation methodologies according to 
their agility. Some metrics are essentially formula-defined functions, whereas others consist of simply 
countable data. Raw data acquired from a variety of sources, including log files, interviews, and surveys, may 
be utilized to derive countable metrics. Countable metrics include the proportion of a task that has been 
accomplished, the ratio of task successes to failures, and the frequency with which technique aids are 
utilized. Calculable metrics are the outcomes of mathematical computations or heuristics utilizing raw 
observational data or countable metrics as inputs. Where Quantity is the proportion of the work done and 
Quality represents the proportion of the objective attained. 
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Table 2. Usability evaluation metric 


Attribute Metric 
Effectiveness Feedback effect 
Quality 
Dynamicity Environmental software adaptability 
Environmental hardware adaptability 
Adaptability 
Time-cost Short turnaround time 
High throughput 
Effort-cost Ease of method effort 


Human effort 
Resources effort 


Learnability Ease of method learning 
Understandability Ease of method understanding 
Results accuracy Incomplete result 


Incorrect result 
Unexpected results issued 


Coverage area Coverage 
Attitude Acceptance of method 
Satisfaction Satisfaction with method 


6. VIEWS OF THE MODEL 

View of usability evaluation: This view is grouped by the selected method's name. It is 
advantageous when developers are familiar with a certain usability evaluation approach (they have heard of 
it) or have been convinced of its usability benefits. This perspective of the model provides the opportunity to 
learn the characteristics of a certain usability evaluation approach. View of development activities: During the 
development process, while developers are trying to find acceptable usability evaluation techniques, they 
utilize this view of the model. To utilize the model for this purpose, developers have to map the model's 
procedures to the exact names of activities regarded by their organization. Then, among the offered ways for 
each type of activity, individuals can choose the ones that best suit their unique goals. 

The offered views in the model are not exclusive. The modeled perspectives are not exclusive. They 
may be used independently, but the model is anticipated to be utilized through a multi-view information 
search. For the purpose of introducing usability evaluation procedures into the development process, 
developers are able to move between views to access the kind of data. For such an activity, the developer will 
select one or more ways based on both the characterisation of the criteria and the information regarding the 
optimal time for method application. The developer may switch to the view by usability evaluation 
techniques in order to review the training standards of the approach and determine if certain members of the 
development team need to attend a lengthy usability training session. We believe that these various 
perspectives of the model provide a varied resource for improving the usability integration attempt. In order 
to give software developers with more guidance, the model is not intended as a tool for the automatic 
development of a custom process add-in, but rather it assists them in the complex work of incorporation. 

Knowing where to inject usability evaluation techniques and activities into the existing process does 
not automatically enable software developers to implement usability practices in their daily work. We include 
a basic reference for each usability evaluation technique in the model, to which developers can consult for 
further information on the method's use. 


7. VERIFICATION OF THE MODEL THROUGH FOCUS GROUP 

The model is verified through focus group technique, which was attended by the domain experts from 
both agile and usability backgrounds. The following sub-sections discuss the implementation of the focus group 
technique, which constitutes the planning, executing, along with analyzing the date and reporting the result [24]. 


7.1. Plan of the focus group 

Thorough planning is needed to effectively implement the focus group technique. In that, five main 
activities were proposed within the plan of the implementation: i) defining the objectives, ii) identifying the 
participants, iii) scheduling the meeting, vi) preparing the materials for the focus group and v) sending 
reminders to the participants. These activities are further elaborated on in the following sub-sections. 


7.2. Define the main objective of the focus group 

Basically, the objective of the focus group is to verify attributes, metrics, measures and scales of the 
proposed model based on certain criteria. These criteria involve comprehensiveness, accuracy, 
understandability and sufficient (see Table 3), which are adapted from previous studies [25]-[27]. 
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Table 3. Descriptions of verification criteria 


Criteria Descriptions 
Comprehensiveness This criterion shows that the attributes, metrics, measures and scales are included in the model. 
Understandability The criterion suggests that the attributes, metrics, measures and scales are decomposed clearly and unambiguously. 


Sufficient The criterion indicates that the attributes, metrics, measures and scales are decomposed to achieve sufficient 
classification. 
Accurateness The criterion indicates that the attributes, metrics, measures and scales are adequately decomposed to achieve 


accurate classification. 


7.3. Participants identification and recruitment 

The participants were selected by using purposive sampling [28]. They were chosen based on 
several common characteristics such as: i) developing software, ii) working in Kuala Lumpur or nearby 
areas, iii) Working with the agile methods, and iv) concerning with usability issues. 


7.4. Meeting schedule 

The suitable meeting places were identified based on the guidelines provided by [29]. The places 
were chosen to be near and suitable for the experts, which include facilities such as a meeting room, 
discussion table, laptops, LCD projector, and other documents. Also, they were provided a pleasant and 
comfortable environment for the experts. Additionally, all appointments were scheduled on Saturday and 
Sunday, which were convenient for the experts. 


7.5. Preparation of the focus group interviews 

Prior to conducting the interview with the experts, the principles of preparing the interview guides 
were adept, whereby the discussion was planned to be started with a general topic, which is the introduction 
of the study. Then, the verification process of the proposed model was continued. Additionally, the materials 
that were used during the interviews were prepared, such as the presentation slides, the proposed model 
materials, and documents for the experts. 


7.6. Remind the experts 

One day before an interview, the expert is reminded about the interview to confirm with them. This 
procedure was conducted to ensure that the experts would not miss the interviews as well as to make them 
feel their importance in attending the interviews. 


7.6. Conducting process of the focus group 

The focus group was conducted on the scheduled day and time. However, two of the participants 
who agreed to come could not attend the session. Thus, only six of the participants turned up to attend the 
session, which is considered a sufficient number of participants for a focus group, the guideline provided by 
[29], [30] were followed. 

Upon arrival at the meeting room, the participants were greeted and friendly contact was established 
in order to create rapport. This was done by having an informal conversation among the participants and 
moderators before the formal discussion begins. They were also served with refreshments. This was intended 
to make the participants feel comfortable and relaxed. On top of that, this enabled the moderators and 
participants to get to know each other. In the formal session, the participants were seated at a U-Shaped 
discussion table to facilitate interactions. They were provided with the materials that were needed for the 
session. They were encouraged to express their experience and points of view freely and spontaneously. 


8. RESULTS FOR THE ATTRIBUTES, METRICS, MEASURES AND SCALES 

In a nutshell, all of the experts agreed that the attributes, metrics, measures and scales are 
comprehensive, understandable, sufficient and accurate (see Table 4). However, they had some comments on 
the understandability, sufficient and accurate of the attributes, metrics, measures and scales. For example, 
Expert A suggested that the metrics of the attributes such as effectiveness are required further clarification, 
while Expert D and G claim that some measures of the metrics are not much enough clear that require more 
description. Meanwhile, Expert B concluded that the measures relevant effort-cost are not enough to measure 
this attribute and he suggested proposing extra effort-cost measures. Lastly, Expert C and D suggested that 
the selection criteria should be organized based on three scales rather than five. However, all suggestions of 
the expert were taken into consideration. Hence, the attributes, metrics, measures and scales were updated 
accordingly as suggested. 
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Table 4. Recapitulates the comments from the knowledge experts 
Expert | Comprehensiveness_ _Understandability Sufficient Accurate 
A Agreed -Suggested that the metrics of the Agreed Agreed 
attributes such as effectiveness 
are required further clarification 
B Agreed -Claimed that the measures Agreed 
relevant effort-cost are not 
enough to measure this 
attribute 
-Suggested to propose extra 
effort-cost measures 
C Agreed Agreed Agreed -Suggested that the 
selection criteria should be 
organized based on three 
scales rather than five 


D Agreed -Some measures of the metrics are Agreed -Suggested that the 
not enough clear that require selection criteria should be 
more description organized based on three 

scales rather five 

E Agreed Agreed Agreed Agreed 

G Agreed -Some measures of the metrics are Agreed Agreed 


not enough clear 


9. CONCLUSION 

This paper presents a model for increasing representation of usability evaluation activities within the 
agile development process by supporting the timely and relevant selection and application of usability 
evaluation methods. The model characterizes each method of usability evaluation according to ten attributes 
that may be of interest to the practitioners in the usability integration endeavor. Further, the model was 
verified in a focus group technique, in which the verification process involved experts from both usability 
and agile backgrounds. Future work can focus on implementing the model in the development environment 
based on practitioner case studies. 
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